2020.2 (latest)

Upgrading

On this page:

UpgradeWizard

Upgrade Wizard for Pronghorn (IAP)

The following steps need to be completed on each instance of IAP. If operating in a clustered environment, please ensure that all instances are updated. Since all instances of IAP attached to a mongoDB will pickup tasks to be worked in a job, having multiple versions of IAP operating on a single MongoDB instance may cause inconsistencies in task processing.

Verify Install Space Exists

  1. SSH into all the IAP servers as admin user. Alternatively, you can sudo to an admin user after logging in.

  2. Check diskspace. Remove older Pronghorn versions as needed.

    df -kh

  3. Make sure <PRONGHORN_HOME> is less than 70% full.

Run the Upgrade Wizard

  1. Run install commands on server. The bin file does not need to be in the <PRONGHORN_HOME> directory.
   cd <PRONGHORN HOME>
   ./itential-<build-id>.bin -h


Example Output

   USER@HOSTNAME pronghorn % ./itential-bundle-5-20193_2019.3.6.linux.x86_64.bin -h
   ********************************************************************************
   Itential Automation Platform (c) 2019 Itential, LLC

   This program will install the Itential Automation Platform. The Itential
   Automation Platform requires access to a server with mongodb installed.

   Installation Help:
   Usage: ./itential-bundle-5-20193_2019.3.6.linux.x86_64.bin [OPTIONS]

   Install / Uninstall / Upgrade:
   -i|--install        Install
   -u|--uninstall      Uninstall
   -p|--upgrade        Upgrade

   Locations:
   -d | --dir <path>       Installation directory. Default: /opt/pronghorn
   -l | --log <path>       Log directory. Default: /var/log/pronghorn
   -c | --custom <path>    Zip file/directory containing custom content (css, apps)
                           Example: /opt/pronghorn/custom-content.tar.gz
   Mongodb:
   --mongohost <host>      Mongodb host address. Default: 127.0.0.1
   --mongoport <port>      Mongodb host port. Default: 27017
   --mongouser <user>      Mongodb username for login
   --mongopass <pass>      Mongodb user password
   --mongodatabase <pass>  Mongodb database name

   Miscellaneous:
   -v | --verbose          Enable verbose output
   -h | --help             Print this help information and exit
   -e | --extract          Extract tar archive and exit
   -y | --unattended       Assume defaults for all prompts & run non-interactively
   ********************************************************************************


  1. Run the following.

    bash ./itential-<build-id>.bin -p

  2. When prompted to extract, type y then press enter. This step may take 3-8 minutes to complete.

Manual Update

Alert This manual update is required only if you did not run the Wizard Update.

  1. Run install commands on server.
   cd <PRONGHORN HOME>
   ./itential-<build-id>.bin -h


Example Output

   USER@HOSTNAME pronghorn % ./itential-bundle-5-20193_2019.3.6.linux.x86_64.bin -h
   ********************************************************************************
   Itential Automation Platform (c) 2019 Itential, LLC

   This program will install the Itential Automation Platform. The Itential
   Automation Platform requires access to a server with mongodb installed.

   Installation Help:
   Usage: ./itential-bundle-5-20193_2019.3.6.linux.x86_64.bin [OPTIONS]

   Install / Uninstall / Upgrade:
   -i|--install        Install
   -u|--uninstall      Uninstall
   -p|--upgrade        Upgrade

   Locations:
   -d | --dir <path>       Installation directory. Default: /opt/pronghorn
   -l | --log <path>       Log directory. Default: /var/log/pronghorn
   -c | --custom <path>    Zip file/directory containing custom content (css, apps)
                           Example: /opt/pronghorn/custom-content.tar.gz
   Mongodb:
   --mongohost <host>      Mongodb host address. Default: 127.0.0.1
   --mongoport <port>      Mongodb host port. Default: 27017
   --mongouser <user>      Mongodb username for login
   --mongopass <pass>      Mongodb user password
   --mongodatabase <pass>  Mongodb database name

   Miscellaneous:
   -v | --verbose          Enable verbose output
   -h | --help             Print this help information and exit
   -e | --extract          Extract tar archive and exit
   -y | --unattended       Assume defaults for all prompts & run non-interactively
   ********************************************************************************


  1. Run the following.

    bash ./itential-bundle-5-20193_2019.3.6.linux.x86_64.bin -e

  2. When prompted to extract, type y then press enter.

  3. Copy over previous Pronghorn (IAP) configs.

   cd itential-bundle-5-20193_2019.3.6
   cp ./properties.json ./properties-orig.json
   cp ../current/properties.json ./properties.json
  1. Copy over previous Pronghorn (IAP) keys.
   cp -R ../current/custom ./
   cp ../current/keys/* keys/

  1. Copy over previous Pronghorn custom folders.

    cp -R ../current/node_modules/@itentialopensource ./node_modules

    This step might fail if the previous version of Pronghorn (IAP) was not using any Itential Open Source services.

  2. Copy over previous Pronghorn (IAP) custom applications. This step may take 3-8 minutes to complete.

    cp -R ../current/node_modules/@<custom app names> ./node_modules

Alert This step is required only if you did not run the Wizard Update.

  1. Change directory to the Pronghorn home directory.

    cd <PRONGHORN_HOME>

  2. Unlink current with rm command. The name of this file will depend on the exact build you were on.

    rm ./current

  3. Create the new link.

    ln -s <IAP 2020.1 Version> current

Migration Scripts

  1. Change directory to the migration script database, then run the scripts.
   cd <PRONGHORN_HOME>
   cd ./current/node_modules/@itential/pronghorn-core/migration_scripts
   node migratePropertiesToDatabase.js

Important Info

  • When prompted to enter (y/n) enter y.
  • Running node migratePropertiesToDatabase.js unattended will automatically respond yes to all prompts, requiring no user input.
  • During an upgrade, a database backup will be triggered via installer script. For instances where IAP is connected via SSL to a Mongo database and Mongo is configured with the requireSSL property, the backup may fail with the following message:
   Failed: error connecting to db server: no reachable servers.
  1. Respond to the migration script prompts to match the deployment environment.
      Retrieving properties.json file...
      Found properties.

      Connecting to Database...
      Are you sure you want to delete the following databases:
      iap_profiles
      service_configs
      (y)es (n)o y
      y
      No service_configs collection detected, skipping.
      No iap_profiles collection detected, skipping.
      RabbitMQ hostname and port are currently set to "localhost:5671". Do you wish to keep these?
      (y)es (n)o y
      y
      Retaining localhost:5671
      Would you like to run WITHOUT SSL?
      (y)es (n)o y
      y
      RabbitMQ credentials currently set to

      username: guest
      password: guest

      Do you wish to keep these?
      (y)es (n)o y
      y
      Retaining guest credentials
      RabbitMQ properties:

      {
         "protocol": "amqp",
         "hostname": "localhost",
         "port": 5672,
         "username": "guest",
         "password": "$ENC84ff82395069f15bdb8e014991d32417818a228dad",
         "locale": "en_US",
         "frameMax": 0,
         "heartbeat": 0,
         "vhost": "/",
         "certPath": "",
         "keyPath": "",
         "passphrase": "guest",
         "caPath": ""
      }

      Backing up properties.json to /opt/pronghorn/itential-1568912783776/properties_9c95343d-51df-41a7-9073-d21f34c840f8.json

      Loading services...
      (node:7345) Warning: Use Cipheriv for counter mode of aes-256-ctr
      Services loaded 8

      Generating init file data...
      Config data generated.

      Generating DB data...
      DB data generated.

      Connection established.

      Creating databases...
      Creating service_configs
      Creating iap_profiles
      Database creation complete.

      Creating IAP database config...
      Collection iap_profiles created!
      IAP database config created.

      Creating service configs entries...
      Collection service_configs created!
      Service configs entries created.

      Cleaning up properties.json...
      Migration complete!

Note: The migration script output above assumes that RabbitMQ is running on the same node as IAP, using default RabbitMQ credentials, and has SSL disabled. For security reasons this RabbitMQ configuration is not recommended for a production environment.

  1. If you have already migrated, a notification will be displayed indicating this.
   Retrieving properties.json file...
   Found properties.

   Already Migrated!

  1. The migration script will migrate much of the /opt/pronghorn/current/properties.json content to the Mongo database.

  2. A backup of the original properties.json file is generated during migration and will appear similar to the following.

   /opt/pronghorn/current/properties_4d37a26d-e5e1-4796-845d-220f09681d65.json
  1. Note any adapter schema errors in startup logs. Errors for adapter schema issues should be fixed in the properties.json file, or by using the System Config editor in the Admin Essentials | Adapters view.

Normalize Groups Scripts

The following procedure is only necessary if upgrading from 2019.3.2 or lower to release version 2019.3.3 or higher. If you have already performed these steps in an update to 2019.3.3 or above, then this procedure is not needed. There is no harm, however, in double-checking to be sure whether or not these steps may apply.

To run the normalizeGroups script:

cd <PRONGHORN_HOME>/current/node_modules/@itential/app-workflow_engine/scripts
npm run index
npm run normalizeGroups

It is very important that you ensure all indexes are setup correctly on Jobs and Tasks. If you are missing any indexes, it may take 15-30 mins to complete the index creation on large databases.

The script will modify each job within the database, so depending on the number of jobs it may take 5 - 15 mins to complete. This is the slowest method because it is run from the IAP server and is only recommended for instances where the jobs collection is moderately sized (low 10K number of records).

Migration Scripts

The following procedure is only necessary if upgrading from 2019.3.2 or lower to release version 2019.3.3 or higher. There is no harm in running these scripts again. Workflows can only be migrated once. If this script is needed to run on the workflow for a second time, the workflow will need to be re-imported into IAP and the script ran again.

To run migration scripts:

cd <PRONGHORN_HOME>/current/node_modules/@itential/app-workflow_engine/migration_scripts
node migrateWorkflows
node migrateJobsAndTasks20192

Compatibility Check Scripts

Automations (within the Automation Studio) no longer allow tasks that export more than one job variable or that attach translations to exported job variables within automations. Workflow Engine will not support running automations with tasks that have translations on your outgoing data, or export more than one job variable as outgoing data. These steps will ensure this issue is resolved.

  1. Run the normalizeGroups script.
   cd <PRONGHORN_HOME>/current/node_modules/@itential/app-workflow_engine/migration_scripts
   node compatibilityCheck20193Maintenance.js
  1. If any automations are discovered to be incompatible, please refer to the Breaking Changes document to resolve them.
   $ cd /opt/pronghorn-core/node_modules/@itential/app-workflow_engine/migration_scripts/
   $ node compatibilityCheck20193Maintenance.js

   incompatibleWorkflow contains incompatible tasks:

   getAdaptersForDevice (508a) Task Summary: exports more than one job variable Reason: Task exports more than one job variable, each of which may or may not have translations attached to it. locateDevices (e7c) Task Summary: exports a job variable with a translation Reason: Task exports a job variable with a translation attached to it.

   locateDevices (37ec) Task Summary: exports more than one job variable with translations Reason: Task exports more than one job variable, each of which may or may not have translations attached to it.

   RunCommand (2aed) Task Summary: exports more than one job var with no translations Reason: Task exports more than one job variable, each of which may or may not have translations attached to it.


Note: For ease of reference, the script output provides the name of the automation (e.g., incompatibleWorkflow), and the taskname (e.g., getAdaptersForDevice and locateDevices).

AG Manager Migration Scripts

This section will help you understand how to run the migration scripts that are necessary in AG_Manager.

  1. Change directory to the migration script database, then run the scripts.
   cd <PRONGHORN_HOME>
   cd ./current/node_modules/@itential/app-ag_manager
  1. Backup the workflows collection before running the scripts. A database backup is strongly recommended before running migration scripts. Please use the following link to learn more: Mongo Export.

Script to update outgoing variable from result to stdOut

This migration script is for automations built prior to release 2020.1 having output variables named result inag_manager.

  1. If the outgoing variable is named result instead of stdOut in AG_Manager tasks, run this script to change it to stdOut.
   cd <PRONGHORN_HOME>
   cd ./current/node_modules/@itential/app-ag_manager
   npm run resultToStdOut
  1. If a database backup exists, enter Y when prompted.
  2. Once script execution is successful, refresh the automation.
  3. Verify the outgoing variable is changed to stdOut.

Script to update query and eval tasks with correct schema reference in app-ag_manager task output

This migration script is for automations built prior to release 2020.1. With the introduction of app-ag_manager in release 2020.1, Itential made a breaking change in the output schema of AG_Manager tasks. As a result, queries built with older schemas are incorrect.

  1. By running this script, all queries in the query and eval tasks containing references to the older schema are changed to the new query task reflecting the new (correct) schema.
   cd <PRONGHORN_HOME>
   cd ./current/node_modules/@itential/app-ag_manager
   npm run pre20201Response
  1. If a database backup exists, enter Y when prompted.
  2. Verify updates to the query and eval tasks.

Script to add env_vars variable in script tasks

This migration script is for automations containing script tasks built prior to release 2019.3. In that release, Itential introduced the env_vars variable in all script executions tasks. As a result, the automations built before 2019.3 would fail to execute.

  1. Before running the script, app-ag_manager needs to be instantiated with an updated pronghorn.json. We further recommend that all Automation Gateway adapters are restarted to generate the updated pronghorn.json and app-ag_manager is instantiated with the latest changes.
   cd <PRONGHORN_HOME>
   cd ./current/node_modules/@itential/app-ag_manager
   npm run addScriptEnv
  1. If a database backup exists, enter Y when prompted.
  2. Verify the automations containing the script tasks can execute successfully.

Script to add groups variable in ag_manager tasks

This migration script is for automations containing Ansible tasks built prior to release 2020.2. In that release, Itential introduced a groups variable in all ag_manager Ansible tasks (i.e., modules/role/playbook). As a result, the automations built before 2020.2 would fail to execute.

  1. Before running the script, app-ag_manager needs to be instantiated with an updated pronghorn.json. We further recommend that all Automation Gateway adapters are restarted to generate the updated pronghorn.json and app-ag_manager is instantiated with the latest changes.
   cd <PRONGHORN_HOME>
   cd ./current/node_modules/@itential/app-ag_manager
   npm run ansibleAddGroups
  1. If a database backup exists, enter Y when prompted.
  2. Verify the automations containing the Ansible tasks can execute successfully.

Script to add cluster support in app-ag_manager

Note: Only run this script if cluster implementation is needed.

  1. What is this clustering implementation all about?

    • If there are multiple IAGs connected to IAP (multiple Automation Gateway adapters) and those IAGs do not have the same version (schema) of modules, roles, and scripts, you will receive a notification that indicates:
      conflict implementation error on app-ag_manager tasks

    • Itential highly recommends having all IAGs with the same Ansible version and modules, roles, scripts with the same schema across all IAGS so that the above error never occurs.

    • If for some reason there is a need to use a certain number of IAGs with different versions, we have implemented clustering through a cluster_name property that is set in the service_config of adapter-automation_gateway.

  2. How does this clustering capability work?

    • Suppose there are 10 IAGs connected to IAP (i.e., 10 Automation Gateway adapters in IAP). If 6 IAGs have the same Ansible version, module and script schemas while the other 4 IAGs have modules with a different schema or Ansible version, we can place 6 IAGs in a cluster named cluster1 (you can use a unique name that's meaningful). All the ag_manager tasks for those IAGs will be changed to {task_name}_cluster1.

    • The other 4 IAGs can be placed in another cluster named cluster2. Similarly for that cluster, the ag_manager task names will be changed to {task_name}_cluster2. With this capability, we have eliminated the conflict error in app-ag_manager.

    • Of note, you only need to implement cluster logic if there are same name modules and scripts in different IAGs with different schema. If all IAGs have the same schema, there is no need to implement cluster logic.

  3. When should we run this migration script?

    • Only run this migration script if there is a conflict implementation error in the app-ag_manager tasks, and you want to use multiple IAGs with different modules or script schemas and have implemented cluster_name properties in the respective Automation Gateway adapters.

    • The cluster_name property is set in adapter-automation_gateway and the adapter must be restarted.

    • Implementation can be verified from the Automation Studio app, where app-ag_manager tasks from the cluster will be seen as {task_name}_{cluster_name}.

  4. What are some limitations of this script?

    • Currently, this script can only migrate old automation app-ag_manager tasks to one cluster immplementation.

  5. To run this script:

   cd <PRONGHORN_HOME>
   cd ./current/node_modules/@itential/app-ag_manager/migration_scripts
   node migrateCluster.js

  1. If a database backup exists, enter Y when prompted.

  2. If running this script for the first time, press Enter as there will not be any tasks with cluster_name.

  3. If changing from one cluster name to another, you must enter the name of the cluster that you want to implement.

    • Enter the name of the cluster you want to rename (i.e., old cluster name).
    • Next, enter the name of cluster that you are changing to (i.e., new cluster name).

  4. All the previous app-ag_manager tasks in old automations will be changed to {task_name}_{new cluster_name}.